Ear insert shape determination

ABSTRACT

There is provided a method of determining a three-dimensional shape of an insert for insertion into an ear. The method includes receiving image data corresponding to a two-dimensional image of an ear, processing the image data to measure at least one biometric feature of the ear, the at least one biometric feature being indicative of a three-dimensional shape of at least part of the ear, and determining a three-dimensional shape of an insert for insertion into the ear by matching said at least one biometric feature with one of a plurality of pre-stored three-dimensional shapes. Each pre-stored three-dimensional shape corresponds to a respective ear.

The present application is a continuation of U.S. application Ser. No.16/958,692, filed Jun. 27, 2020, which is a US National Stage Entry ofPCT/EP2018/086588, filed Dec. 21, 2018, which in turn claims priorityfrom GB 1722295.1, filed Dec. 29, 2017, the entirety of all of which areexplicitly incorporated herein by reference.

TECHNICAL FIELD

The present invention relates to methods, apparatus and systems fordetermining a three-dimensional shape of an insert for insertion into anear from a two-dimensional image of the ear. The invention hasparticular, but not exclusive, relevance to the manufacture of an earbudwhose shape is customised to fit in an ear.

BACKGROUND

Systems for manufacturing custom earbuds are known. Generally, suchsystems either utilise a mould or specialist equipment to determine ashape for insertion into the ear. For example, it is known to producecustom-fitted in-ear headphones for a person that are typically morecomfortable, and less likely to fall out of the ear, than standardisednon-custom earbuds. There is, however, a desire to develop analternative methodology for determining the shape of an ear insert whichdoes not require utilising a mould or specialist equipment, therebymaking the process less expensive and alleviating the logistical problemof bringing together the subject person and the specialist equipment orsomeone capable of taking a mould.

US patent application no. 2010/0296664 discusses a system for providingearpieces which utilises a non-contact 3D scanner to generatethree-dimensional data for a customer's ear. Such a non-contact 3Dscanner is the type of specialist equipment that introduces cost andlogistical issues as discussed above. US 2010/0296664 acknowledges thatthere are algorithms that will try to infer three-dimensional data froma two-dimensional image of an ear, but notes that these are subject tomany errors and therefore can be inaccurate and unreliable.

SUMMARY

Aspects of the invention are set out in the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the present invention will now be described, byway of example, with reference to the accompanying Figures in which:

FIG. 1 shows schematically a system according to an exemplary embodimentof the present invention;

FIG. 2 shows schematically a method for manufacturing an earbud usingthe system of FIG. 1 ;

FIG. 3 shows a fitting card at three different orientations;

FIG. 4 shows schematically a method for extracting features from atwo-dimensional image;

FIG. 5A shows images of an ear with biometric features that are measuredby the system of FIG. 1 indicated;

FIG. 5B shows an image of an ear with a set of biometric featuresindicated;

FIG. 6 shows schematically a method of training a machine learningclassification algorithm;

FIG. 7 illustrates three-dimensional data for an ear shape;

FIG. 8 shows schematically a system for producing a database ofpre-stored ear shapes, and for matching two-dimensional images of earsto ear shapes in that database;

FIG. 9 shows schematically a system for populating a database withthree-dimensional ear shapes;

FIG. 10 shows schematically a further system for populating a databasewith three-dimensional ear shapes.

FIG. 11 is a flow chart illustrating a method of determining athree-dimensional shape for an insert to be inserted into an ear;

FIG. 12 shows schematically an apparatus for determining athree-dimensional shape for an insert to be inserted into an ear; and

FIGS. 13A and 13B show schematically systems according to embodiments ofthe present invention.

DETAILED DESCRIPTION

Embodiments of the present invention involve determining the shape of athree-dimensional insert for insertion into an ear from atwo-dimensional image of the ear. This allows a custom-fitted earbud tobe manufactured based on a photograph of a user's ear, for examplecaptured with a smartphone, without requiring any three-dimensionalscanning of the user's ear and thus does not require the specialisedscanning equipment that such an approach would entail. Furthermore,because the user can capture the photograph with their own smartphone,there is no requirement for the user to visit a scanning location inorder to determine the ear shape. This increases the convenience to theuser, as well as reducing the costs of determining the ear shape andthereby reducing the costs of manufacturing custom-fitted earbuds.

The shape of the ear insert, such as an earbud, is customised forinsertion into at least part of the auricle and the ear canal of theear. In particular, the ear insert is shaped to match, within varioustolerances, the shapes of at least some of the Concha Cavum, ConchaCymba, Antitragus, Tragus, Tragus Notch and the Ear Canal, and to sitover the Helices Crus.

FIG. 1 shows schematically a system according to an exemplary embodimentof the present invention. The system includes a user's smartphone 105, aserver 120, a database 130, and a 3D printing system 135. A smartphoneis a mobile telephone that, in addition to being arranged to performconventional audio communications, has processing circuitry that iscapable of executing downloaded software applications, commonly referredto as apps.

The smartphone 105 includes, among other functionalities, a camera 110and an earbud app 115. While the camera 110 is integral to thesmartphone 105, it will be appreciated that the earbud app 115 willtypically be downloaded onto the smartphone 105 from an “app store”,although alternatively the earbud app 115 could be, for example a webapp. In other examples, the role of the smartphone 105 may instead beperformed by, for example, a desktop computer, a laptop computer, atablet computer, a digital camera, or any other suitable device capableof capturing, processing and/or transmitting image data in accordancewith the present method.

The earbud app 115 guides a user through a process for obtaininginformation needed for the manufacture of an earbud, and then sends theobtained information to a remote server 120. This transmission may beconducted via a wireless telecommunications network such as WidebandCode Division Multiple Access (WCDMA) or Long Term Evolution (LTE), oralternatively may be transmitted over the internet using Wi-Fi or awired connection.

The server 120 stores matching routine 125 which matches atwo-dimensional image of an ear with one of a plurality ofthree-dimensional ear shapes that are stored in a database 130. Thethree-dimensional ear shapes each include at least portions of the earcanal and/or auricle. Although the database 130 is shown as beingseparate from the server 120 in FIG. 1 , it may alternatively be storedby the server 120. This operation of the matching routine 125, which isdescribed in more detail below, generally involves extracting particularanatomical features and making associated measurements, such asdistances between specific points on the ear, from the image andidentifying the three-dimensional ear shape stored in the database 130that corresponds to the closest match to the extracted anatomicalfeatures taking into account the various tolerances for themeasurements.

In this embodiment, the server 120 is connected to a three-dimensionalprinting, or additive manufacturing, system 135. The three-dimensionalprinting system 135 produces an earbud based on the three-dimensionalear shape identified by the matching routine 125 and the requestedearbud parameters. The resultant earbud is then shipped to the user. Inthis embodiment, the earbud is shipped to the user as part of anearphone.

The system of FIG. 1 therefore allows the user to obtain a custom-fittedearbud based only on a two-dimensional image captured with theirsmartphone 105, without having to perform any three-dimensional scanningof their ear. Rather than trying to infer three-dimensional data fromthe two-dimensional image of the ear, the system uses features extractedfrom the two-dimensional image to identify a match within a database ofthree-dimensional ear shapes.

FIG. 2 shows schematically a method 200 for manufacturing an earbudaccording to an embodiment of the present invention. In this example,the method can be implemented within the system described above inrelation to FIG. 1 .

Following opening by a user, the earbud app 115 displays, at 205, aguide to the user. This guide includes prompts for the user to input allthe information required for the manufacture of a custom earbud.Included in this, the earbud app 115 prompts, at 210, the user tocapture an image of their ear using the camera 110. The earbud app 115provides detailed instructions regarding how the user should capture animage of their ear. For example, the earphone app 115 informs the userof the required lighting conditions, the required distance that thecamera 110 should be away from the ear, and the correct orientation ofthe camera 110 with respect to the ear.

In this example, absolute sizes of the extracted features are determinedbased on an object of known spatial dimension included in the capturedimage. In particular, the earbud app instructs the user to hold a coin(or other object of known size) next to their ear, based on which thescale of the captured image can be determined.

Other information obtained by the earbud app 115 includes, for example,parameters of the desired earbuds such as colour, style, cordless vs.corded, speaker specifications or aesthetic design features, and paymentdetails.

The earbud app 115 then prompts, at 215, the user to confirm thepurchase of the earbud. Following this confirmation, the earbud app 115transmits the obtained information to the server 120.

The server 120 verifies, at 220, that the quality of the image of theear is suitable for the matching operation that is described in moredetail below. Examples of quality criteria include: a lack of occlusionof the ear for example by the user's hair, sufficient lighting of theimage, the entire ear being visible in the image, and the ear beingparallel with the focal plane of the camera. If the quality isinsufficient, then the server 120 sends a signal to the earbud app 115to request a replacement image. In other embodiments, the verificationis performed by the earbud app.

The server 120 then extracts, at 225, from the two-dimensional image ofthe ear features of the ear that are representative of thethree-dimensional ear shape. As described in more detail below, thesefeatures are typically measurements of anatomical features of the ear.As the anatomical features must be visible in the two-dimensional image,they are typically features of the external auricle of the ear.

The server 120 then matches, at 230, the ear with one of a plurality ofpre-stored three-dimensional ear shapes, stored in a database. Asdescribed in more detail below, the matching comprises determining whichpre-stored shape of the plurality most closely matches the capturedtwo-dimensional image of the ear, based on the aforementioned extractedfeatures, taking into account the various tolerances for themeasurements.

In particular, in this embodiment each of the pre-storedthree-dimensional shapes is stored in association with a two-dimensionalprojection of the three-dimensional shape and biometric features derivedfrom that two-dimensional projection. By comparing the biometricfeatures extracted from the received two-dimensional image with thebiometric features associated with each pre-stored three-dimensionalshape, the closest match can be determined.

An earbud is then manufactured, at 235, according to thethree-dimensional ear shape identified in step 230. This earbud is thenshipped to the user.

In the example described above, the earbud app 115 instructs the user toinclude an object of known spatial dimension in the captured image ofthe ear, from which the absolute sizes of the extracted features aredetermined. In another example, a system for determining a shape of aninsert for insertion into an ear includes, in addition to the componentsof FIG. 1 , a fitting card for determining a scaling and orientation ofa captured image of an ear. FIGS. 3 a, b, and c show examples of afitting card 300 at three different orientations. The fitting card 300includes a central hole 305 through which a user can extend his or herear, or through which the user's ear is visible when the fitting card isplaced against the side of the user's head. The fitting card 300 alsoincludes gridlines 310, which in this example are evenly spaced andmutually perpendicular, thus forming a rectilinear grid. In otherexamples, a fitting card may include additional or alternative featuresto those shown in FIG. 3 , or may omit certain features such as thegridlines 310.

FIG. 4 shows schematically a method 400 for processing an image of anear in accordance with an embodiment of the present invention. Prior tothe method of FIG. 4 being performed, the earbud app 115 running on thesmartphone 105 instructs the user to place the fitting card 300 againstthe side of his or her head, with his or her ear extending through thehole. After capturing an image of the ear of the user with the fittingcard in place, the smartphone 105 sends the captured image to the server120. The server 120 receives, at 405, the captured image from thesmartphone 105.

The server 120 detects, at 410, the fitting card 300 in the receivedimage. In this example, the fitting card 300 is detected using standardimage processing techniques to detect the gridlines 310. The inclusionof gridlines and/or other distinctive features on the fitting card 300allows the server 120 to detect the fitting card 300 reliably. In otherexamples, an object detection routine, for example using a trainedneural network or other machine learning algorithm, may be used todetect a fitting card.

The server 120 determines, at 415, a scaling of the image using thedetected fitting card 300. In this example, the server 120 uses thegridlines 310 to determine the scaling of the image, but in otherexamples, other features of the fitting card 300 may be used, forexample the size of the hole 305 or the overall size of the fitting card300.

The server 120 determines, at 420, an orientation of the fitting card300 using the gridlines 310. In this example, determining theorientation involves measuring sizes of the regions delimited by thedetected gridlines 310. For example, in the orientation of FIG. 3 a ,the regions delimited by gridlines 310 towards the left of the fittingcard 300 appear smaller that the regions delimited by gridlines 310towards the right of the fitting card 300. In the orientation of FIG. 3b , the regions delimited by gridlines 310 appear approximately equal insize over the extent of the fitting card 300. In the orientation of FIG.3 c , the regions delimited by gridlines 310 towards the left of thefitting card 300 appear larger that the regions delimited by gridlines310 towards the right of the fitting card 300. In other examples, theorientation of the fitting card 300 may be determined by measuringangles of the gridlines 310, and/or by measuring relative spacings ofthe gridlines 310. The determined orientation may be represented as oneor more numbers corresponding to one or more angles, including forexample an angle of the fitting card around an axis parallel to thevertical gridlines 310 of the fitting card. In a further example, theorientation of the fitting card is classified as “right”, correspondingto an orientation as shown in FIG. 3 a , “straight”, corresponding to anorientation as shown in FIG. 3 b , and “left”, corresponding to anorientation as shown in FIG. 3 c.

In one example, the determined orientation is represented as threenumbers, corresponding to angles of rotation of the fitting card aboutthree mutually perpendicular axes (for example, a first axis passingthrough the camera 110 and the centre of the hole 305, a second axisperpendicular to the first axis and having a predetermined rotationabout the first axis with respect to the camera 110, and a third axisperpendicular to the first axis and the second axis). It will beappreciated that the scaling and orientation may be determined in asingle step.

Having determined the scaling and orientation, the server 120 verifies,at 425, whether the image is suitable for matching. In this example,verifying that the image is suitable for matching includes determiningthat the scaling of the image is within a predetermined acceptablerange, and accordingly that the camera 110 was neither too far from, nortoo near to, the ear of the user at the time that the image wascaptured. Verifying that the image is suitable for matching furtherincludes determining that the orientation of the fitting card 300 isacceptable. For example, where the orientation is determined as one ormore numbers corresponding to one or more angles, each of the anglesmust be within a respective predetermined acceptable range for theserver 120 to verify that the image is suitable for matching. In aspecific example, for an image of a left ear of a user, the orientationsof the fitting card 300 shown in FIG. 3 a and FIG. 3 b are determined tobe acceptable, whereas the orientation of the fitting card 300 shown inFIG. 3 c is determined not to be acceptable. For a left ear, the “left”orientation of FIG. 3 c corresponds to the camera 110 being in aposition in front of the ear of the user, from which certain featureswithin the ear may be hidden. By contrast, the “right” orientation ofFIG. 3 a corresponds to the camera 110 being in a position behind theear of the user, from which the features may be visible.

If the image is not verified to be suitable for matching, the server 120sends a signal, at 430, to the earbud app 115 to request a replacementimage.

If the image is verified to be suitable for matching, the server 120extracts, at 435, the features of the ear that are representative of thethree-dimensional ear shape, as described in more detail hereafter. Inthis example, the extracted features of the ear are based on a set ofdetected anatomical points.

The server applies, at 440, the scaling determined at 415 to theextracted features. In this example, applying the scaling involvesconverting distances between the determined anatomical points frompixels to millimetres.

The server adjusts, at 445, the extracted features to take into accountthe orientation of the fitting card 300 determined at 445. For example,depending on a determined angle of the fitting card 300 about a verticalaxis, the extracted features may be scaled in the horizontal directionaccording to a predetermined rule, such that for any determined angle,the extracted features can be meaningfully matched with correspondingfeatures associated with a database of three-dimensional ear shapes.

In other examples, the fitting card 300 may be omitted, and adjustingthe extracted features to take account of the orientation of the imagemay be performed using image registration techniques, for example usinga neural network or other supervised learning algorithm trained using aset of images of ears captured at different, known, orientations. Inother examples, all or part the method of FIG. 4 may be performed by thesmartphone 105.

In the method of FIG. 4 , the server 120 processes an image captured bythe camera 110 to extract features for a matching operation. In otherexamples, the smartphone 105 may determine the distance to, and/ororientation of, the user's ear with respect to the camera in real timeor near real time, such that the earbud app 115 may automatically detectwhen the user's ear is at an acceptable distance and/or orientation, andautomatically capture an image of the ear or signal to the user tocapture an image of the ear. In one example, the distance andorientation are detected using real time object detection to identify afitting card such as fitting card 300.

As noted above, the operation for matching an image of a user's ear withone of a plurality of pre-stored three-dimensional ear shapes is basedon anatomical features of the user's ear. The dimensions of theanatomical features are representative of the three-dimensional shape ofat least part of the user's ear, for example including a part of theauricle and a part of the ear canal. Some examples of such features willnow be described with reference to FIG. 5 . FIG. 5 shows several imagesof ears 505-535, with example features shown in black lines.

One such feature, shown in image 505, is a curvature of the helix of theear, for example expressed as the relative length of the various linesshown in the image 505, each of which run from a predefined point on theFossa triangularis of the ear to various points on the helix of the ear.

Other features include measurements, for example side lengths and/orareas, of various triangles defined by predefined anatomical points ofthe ear. Examples of such points include points on the helix, Fossatriangularis, lobe, intertragic notch, antihelix, tragus and antitragus.Images 510-535 show various such triangles. Specifically:

-   -   image 510 shows a triangle formed by points on the helix, Fossa        triangularis and lobe;    -   image 515 shows a triangle formed by points on the Fossa        triangularis, intertragic notch and antihelix;    -   image 520 shows a triangle formed by points on the antihelix,        Fossa triangularis and tragus;    -   image 525 shows a triangle formed by points on the tragus,        antitragus and intertragic notch;    -   image 530 shows a triangle formed by points on the Fossa        triangularis, antihelix and tragus; and    -   image 535 shows a triangle formed by points on the tragus,        intertragic notch and antihelix.

The points are identified in the image using a machine learningclassification algorithm, following which measurements of triangles suchas those described above are determined. The classification algorithm istrained on a set of images of ears for which the above-mentioned pointsare known.

FIG. 5B shows an example of a set of anatomical features representativeof a three-dimensional shape of at least part of a user's ear. In thisexample, the set of features includes a triangle formed by points on theFossa triangularis, intertragic notch and antihelix, corresponding tothe feature described above with reference to image 515. The set offeatures further includes the shape of the antihelix, which ischaracterised by a set of of lines extending between a point on thetragus and five respective points on the antihelix, and a further lineextending between the two lowest of the five respective points. Thefeatures shown in FIG. 5B are based on eight predefined anatomicalpoints, though it is envisaged that alternative sets of features may beused in some embodiments, and these alternative sets may be based onmore or fewer than eight predefined anatomical points.

FIG. 6 shows schematically an exemplary method 600 performed by acomputer system to train a machine learning classification algorithm fordetecting anatomical points of an ear.

The computer system receives, at 605, a set of training images in whichthe anatomical points have been labelled by hand. In this example,labelling the anatomical points by hand involves a human useridentifying by eye each of the anatomical points within the image andusing a cursor to tag and label the identified points accordingly. Dataindicative of the co-ordinates of the labelled points within eachtraining image are stored as an additional data layer in associationwith that training image. The set of training images may be captured ata variety of different orientations and in a variety of differentlighting conditions.

The computer system artificially augments, at 610, the set of trainingimages based on, for example, contrast, brightness, scale, andorientation. Augmenting the set of training images involves generatingadditional training images, referred to as artificial training images,by processing the original training images received at 605. Augmentingthe set of training images based on contrast and/or brightness includesvarying the contrast and/or brightness of the images in the original setto generate artificial training images that correspond to the originalimages but have varying levels of contrast and/or brightness. Augmentingthe set of training images based on scale includes scaling the images bypredetermined increments to generate artificial training images thatcorrespond to the original images but at different scales. Augmentingthe set of training images based on orientation includes, for example,rotating the images about an axis perpendicular to the planes of theimages, to generate artificial training images that correspond to theoriginal1 images, rotated by different angles. The computer systemtrains, at 615, the machine learning classification algorithm with theartificially augmented set of training images. In a specific example,the machine learning classification algorithm includes a convolutionalneural network (CNN), and training the machine learning classificationalgorithm involves passing the artificially augmented set of trainingimages through the CNN and performing backpropagation followed bygradient descent to update parameters of the CNN. Training the machinelearning classification algorithm with the artificially augmented set oftraining images improves the ability of the machine learningclassification algorithm to classify images captured in differentlighting conditions and at different angles and distances. Furthermore,the earbud app 115 may be installed on a variety of differentsmartphones, which may have different cameras and/or cameraconfigurations from each other, and training the machine learningclassification algorithm with the artificially augmented set of trainingimages improves the reliability of the algorithm when applied to imagescaptured by a range of different smartphones.

In addition to processing an image captured by the smartphone 105 toextract biometric features of a user's ear, in the present embodimentthe server 120 performs a further machine learning classificationroutine to identify one or more predetermined anomalous ear shapefeatures. One such anomalous ear shape feature is a closed cymba.Another such anomalous ear shape feature is a pronounced ridge in theear, for example corresponding to a pronounced crus helix. It has beenobserved that such anomalous ear shape features can lead to poor fittingand/or comfort of an earbud manufactured according the present method.Accordingly, if one or more anomalous ear shape features is detected,the server 120 sends a signal to the smartphone 105, causing thesmartphone 105 to inform the user that the method of determining theuser's three-dimensional ear shape has been unsuccessful, and thatspecialist equipment is required for determining a shape of an earinsert for the user.

In the present embodiment, indications of the above-described featuresare stored for each of the pre-stored three-dimensional ear shapes, suchthat each measurement is stored for each pre-stored three-dimensionalear shape. As described in more detail below, an initial set of thepre-stored three-dimensional ear shapes may be produced bythree-dimensionally scanning ears. The measurements may be extracteddirectly from the scans or, alternatively, from two-dimensional imagescaptured in addition to the scans.

The matching operation referred to above includes comparing the measuredfeatures from the captured ear image with the biometric features storedfor each pre-stored three-dimensional ear shape. This includesdetermining whether some of the measurements fall within presettolerances. For example, particular measurements corresponding to earbuddimensions that are critical for user comfort may have tighter presettolerances than other dimensions that are less critical for usercomfort. As a particular example, the dimensions of the cymba of the earhave a high impact on the comfort of an earbud. Tighter presettolerances are accordingly applied to measurements that are more closelycorrelated with the dimensions of the cymba. Subject to constraintsbased on tolerances as described above, the captured ear image ismatched with the three-dimensional ear shape for which the featurescorrespond most closely with the measured features from the captured earimage. This matched three-dimensional ear shape is deemed to be theclosest match, of the pre-stored three-dimensional ear shapes, to theear of which the image was captured. A pre-stored three-dimensionalshape may be rejected if one or more of the features of the pre-storedshape falls outside of preset tolerances, even if that pre-stored shapewould otherwise have been deemed the closest match.

A method for obtaining the pre-stored three-dimensional ear shapes, anddetermining the relevant features thereof, will now be described withreference to FIG. 7 .

In order to obtain an initial set of three-dimensional ear shape, an earis three-dimensionally scanned to produce data 705 representing thethree-dimensional ear shape. For example, a user may be incentivised tovisit a scanning kiosk by offering a premium service, or a pricediscount, relative to obtaining custom-fitted earbuds based on atwo-dimensional image.

The scanned ear shape 705 is then rotated in order to produce a rotatedversion 710 that is oriented with an x-y plane. Rotating scanned shapesto the same x-y plane in this manner allows all scanned ear shapes to beconsistently analysed.

Cross sectional layers 715 are then determined, corresponding to a“scanning” of a horizontal plane down the rotated ear shape 710. Thecombined cross sectional layers 415 thereby provide a systematicdescription of the geometry of at least part of the three-dimensionalauricle and ear canal. The cross sectional layers 715 may be used by the3D printing system 135 to perform additive manufacturing of an earinsert, though in other examples the step of determining cross sectionallayers may be omitted, and a three-dimensional ear shape may betransmitted directly to a 3D printing system for manufacture of acorresponding ear insert.

The rotated ear shape 710 is used to determine a two-dimensionalprojection 720 of the three-dimensional ear shape. The two-dimensionalprojection 720 is a two-dimensional image corresponding to a view of theear canal in which features representative of the shape of the ear arevisible. In the present example, the two-dimensional projection 720 is agreyscale image.

The two-dimensional projection 720 is used for matching with atwo-dimensional image of a user's ear, for example captured using themethod 400 of FIG. 4 . In order to use the two-dimensional projection720 for matching, anatomical features such as those described above withreference to FIG. 5A are extracted from the two-dimensional projection720. In this example, the anatomical features correspond to thosedescribed above with reference to FIG. 5A, and are based on predefinedanatomical points, which are identified in the two-dimensionalprojection 720 using a machine learning classification algorithm. Themachine learning classification algorithm is trained using a similarroutine to that described above with regard to FIG. 6 , with a trainingset being augmented based on contrast and orientation.

In the present embodiment, the two-dimensional projection is stored inassociation with the three-dimensional data. When a given ear shape ismatched to be used to manufacture an earbud using the correspondingbiometric features derived from a two-dimensional projection, theassociated three-dimensional data can be used to define the shape of theearbud to be produced.

In the example described above, two-dimensional anatomical featurescorresponding to a pre-stored three-dimensional ear shape are extractedfrom a two-dimensional projection of the three-dimensional ear shape. Inother examples, anatomical features may be extracted directly from athree-dimensional ear shape, without first generating a two-dimensionalprojection of the three-dimensional ear shape. In addition to thethree-dimensional scanning of an ear discussed above, alternativemethods may be used to obtain the three-dimensional ear shapes, forexample scanning a physical mould of an ear. The method of extractingthe anatomical features described above is agnostic to the method bywhich the ear shapes are obtained, allowing an extensive database ofpre-stored ear shapes and associated anatomical features to be built upfrom a range of sources.

For each pre-stored three-dimensional ear shape obtained as describedabove, multiple scaled versions may also be stored. For example,versions scaled uniformly by increments of 5% or 10% may be stored. Inorder to obtain scaled versions, a three-dimensional scaling algorithmis applied to the three-dimensional data 705, and the method describedwith reference to FIG. 7 is applied to generate correspondingtwo-dimensional projections and corresponding biometric features.Storing scaled versions of the three-dimensional ear shapes, along withthe corresponding biometric features, increases the chances of a matchbeing obtained during the matching operation.

As an alternative to extracting features from a two-dimensionalprojection of the three-dimensional ear shape, in some embodiments anactual two-dimensional picture of the ear is stored in association withthe three-dimensional data and the biometric features for that ear aremeasured using that two-dimensional picture. When a given ear shape ismatched to be used to manufacture an earbud using the correspondingbiometric features derived from a two-dimensional picture, theassociated three-dimensional data can be used to define the shape of theearbud to be produced. A system for both producing a database ofpre-stored ear shapes, and matching two-dimensional images of ears toear shapes in that database, will now be described with reference toFIG. 8 .

A scanning centre 805, for example a kiosk, can be visited by a user inorder to receive a three-dimensional scan of their ears. As noted above,a user may be incentivised to use the scanning centre 805 by offering apremium service, such as additional earbud features, for using thescanning centre 805. Alternatively or additionally, a price discount maybe offered to users who use the scanning centre 805. The user's ears arethree-dimensionally scanned as described above in relation to FIG. 7 ,and the resulting ear shapes, along with details of the measurements ofthe features such as those described above in relation to FIG. 5A, arestored in a database 810. Depending on how the three-dimensional earshapes are obtained, some of the three-dimensional ear shapes may bestored in association with a two-dimensional projection derived from thethree-dimensional shape, while other three-dimensional ear shapes may bestored in association with a two-dimensional picture of the ear capturedalongside the three-dimensional shape. In addition to being stored inthe database 810, the scanned three-dimensional ear shapes aretransmitted to a manufacturing centre 815 which produces custom-fittedearbuds for the user, for example by three-dimensional printing.

Over time, as users use the scanning centre 805, the database of 810 isexpanded to include a large number of three-dimensional ear shapes.

A different user uses a smartphone 820 to capture images of their ears.These images are transmitted to a server 825 which matches the imageswith three-dimensional ear shapes stored in the database 810, asdescribed in more detail above. The server 825 then transmits details ofthe matched three-dimensional ear shapes to the manufacturing centre815, which produces earbuds for the user according to the matchedthree-dimensional ear shapes.

If the matching is unsuccessful, because none of the ear shapes storedin the database 810 provide a suitable match to the captured ear images,the server 825 instructs the smartphone 820 to request that the uservisit the scanning centre 805 to receive a three-dimensional scan oftheir ears, with the resulting three-dimensional ear shapes, along withdetails of the measurements of the features such as those describedabove in relation to FIG. 5A, being stored in a database 810. The usermay be incentivised to do this for example by offering a discountedprice. If the matching is successful, but the resulting manufacturedearbuds do not adequately fit the ears of the user, then the user canalso visit the scanning centre 805 to receive a three-dimensional scanof their ears, with the resulting three-dimensional ear shapes, alongwith details of the measurements of the features such as those describedabove in relation to FIG. 5A, being stored in a database 810. It will beappreciated that as the number of ear shapes stored by the databaseincreases, the likelihood of a successful match and a successful fitincreases.

FIG. 9 shows schematically a system for populating a database 905 withthree-dimensional ear shapes.

A user visits a scanning centre 910, similar to the scanning centre 805of FIG. 5A8. Three-dimensional scans 915 are produced of the user'sears. The scans 925 are stored in a three-dimensional scan data store920 of the database 905. A trimming module 923 of a server 935optionally performs a “trimming” operation, in which thethree-dimensional ear shapes are modified in order to be used tomanufacture an earbud. For example, the surface may be smoothed, andbumps or other features that are not desired to be in the manufacturedearbud may be “trimmed”, or removed. It is noted that this operation mayalternatively be performed at the scanning centre 910, and for certaintypes of scan, may not be necessary at all.

A projecting module 924 of the server 935 performs a projectingoperation, as described above, to generate two-dimensional projectionsof the stored three-dimensional ear shapes. The projecting operationincludes orienting the image as described above with reference to FIG. 7. The two-dimensional projections are stored in a two-dimensionalprojection store of the database 905.

The server 935 extracts two-dimensional features from thetwo-dimensional projections in a feature extraction module 940. Thesefeatures include measurements defined by anatomical points of the ear,such as those described above in relation to FIG. 5A. The extractedfeatures are stored in a two-dimensional feature database 945 of thedatabase 905, such that they can be associated with their correspondingthree-dimensional ear shapes stored in the three-dimensional scan datastore 920.

The server 935 later receives captured two-dimensional images of theears of a different user, for example via the smartphone 820 of FIG. 8 ,and matches these images with the extracted features. When a match isdetermined, the corresponding three-dimensional ear shape is identifiedand transmitted to a manufacturing centre such as the manufacturingcentre 815 of FIG. 8 , based on which the manufacturing centremanufactures earbuds.

FIG. 10 shows schematically an alternative system for populating adatabase 1005 with three-dimensional ear shapes. The system of FIG. 10is equivalent to the system of FIG. 9 , except that in the scanningcentre 1010 of FIG. 10 , two-dimensional photographs 1025 are alsocaptured of the users' ears, and stored in a two-dimensional photographdata store 1030 of the database 1005. The server 1035 extractstwo-dimensional features from the two-dimensional photographs in afeature extraction module 1040. In this embodiment, the server 1035 doesnot include a projecting module. Further embodiments may include acombination of features of FIGS. 9 and 10 , such that some images arestored alongside two-dimensional photographs, and other images arestored alongside two-dimensional projections.

FIG. 11 shows schematically a method 1100 of determining athree-dimensional ear shape for an ear.

The method 1100 has a step 1105 of receiving image data corresponding toa two-dimensional image of the ear. As described in more detail below,the two-dimensional image can be captured by a user, for example with asmartphone or other device with a camera.

The method 1100 has a step 1110 of processing the image data to measureat least one biometric feature of the ear. As described in more detailbelow, these biometric features are particular measurable features,visible in the two-dimensional image, that are representative of thethree-dimensional ear shape.

The method 1100 has a step 1115 of determining the three-dimensional earshape for the ear by matching the above-mentioned biometric featureswith one of a plurality of pre-stored three-dimensional ear shapes. Inthis manner, the biometric features are matched with the pre-stored earshapes in order to identify a particular pre-stored shape thatcorresponds most closely to the captured two-dimensional image. Theidentified pre-stored three-dimensional shape can then be assumed todescribe the three-dimensional shape of the ear. A custom-fitted earbudcan then be manufactured according to the identified pre-storedthree-dimensional shape.

FIG. 12 shows schematically an apparatus 1200 for determining athree-dimensional ear shape for an ear, for example my implementing themethod described above in relation to FIG. 11 . The apparatus 1200 mayfor example be a server.

The apparatus 1200 has a receiving module 1205 configured to receiveimage data corresponding to a two-dimensional image of the ear, forexample from a user's smartphone as described above.

The apparatus 1200 has a measuring module 1210 configured to process theimage data to measure at least one biometric feature of the ear, the atleast one biometric feature being representative of thethree-dimensional ear shape. The biometric features may for example bemeasurements defined by anatomical points of the ear, such as thosedescribed above in relation to FIG. 5A.

The apparatus 1200 has a matching module 1215 configured to determinethe three-dimensional ear shape for the ear by matching said at leastone biometric feature with one of a plurality of pre-storedthree-dimensional ear shapes. The plurality of pre-storedthree-dimensional ear shapes may for example be stored in a database asdescribed in more detail above.

Example systems including the apparatus 1200 will now be described withreference to FIGS. 13A and 13B.

Referring to FIG. 13A, a system 1300 includes the apparatus 1200 and adatabase 1305. The database 1305 stores the pre-stored three-dimensionalear shapes. The apparatus 1200 uses the biometric features measured froma captured image to identify the pre-stored three-dimensional ear shapewhich provides the best match as described in more detail above.

Referring to FIG. 13B, a system 1310 includes the apparatus 1200 anddatabase 1305 as shown in FIG. 13A. Furthermore, the system 1310includes an image capture device 1310, such as a smartphone including acamera. The image capture device 1310 is communicatively coupled to theapparatus 1200. The image capture device 1310 is configured to capturethe two-dimensional image of the ear, confirm that the image meetspredefined quality criteria as described in more detail above and,responsive to the confirming, process the two-dimensional image toproduce the image data and transmit the image data to the receivingmodule of the apparatus 1200.

As discussed above, the three-dimensional shape for an insert istypically a trimmed form of a three-dimensional shape that wouldprecisely engage the ear. It will be appreciated that the pre-storedthree-dimensional shapes in the database corresponding to different earscould be stored untrimmed or trimmed. In either case, thethree-dimensional shape of the insert can be determined from thepre-stored three-dimensional shape.

The above embodiments are to be understood as illustrative examples ofthe invention. It is to be understood that any feature described inrelation to any one embodiment may be used alone, or in combination withother features described, and may also be used in combination with oneor more features of any other of the embodiments, or any combination ofany other of the embodiments. Furthermore, equivalents and modificationsnot described above may also be employed without departing from thescope of the invention, which is defined in the accompanying claims.

The invention claimed is:
 1. A method of producing an ear insert, theinsert having a three-dimensional shape which is derived from image datacorresponding to a two-dimensional image of an ear, the methodcomprising: receiving, at a server from a remote device, the image datacorresponding to the two-dimensional image of the ear; scaling andre-orienting, at the server, the image data, to be within apredetermined acceptable range for the server to determine whether theimage data satisfies an image quality criteria suitable for a matchingoperation; and when the image data does not satisfy the image qualitycriteria, rejecting the image data and transmitting to the remote devicefrom the server a request for replacement image data; extracting, at theserver, particular anatomical features and making associatedmeasurements from the image data to obtain at least one biometricfeature of the ear, the at least one biometric feature of the ear beingrepresentative of a three-dimensional shape of at least part of the ear;and determining, at the server, the three-dimensional shape by matchingsaid at least one biometric feature of the ear with one of a pluralityof pre-stored three-dimensional ear shapes, wherein each pre-storedthree-dimensional ear shape corresponds to a respective ear.
 2. Themethod according to claim 1, wherein the at least one biometric featureof the ear is matched to a two-dimensional ear representationcorresponding to said one of the plurality of pre-storedthree-dimensional ear shapes.
 3. The method according to claim 2,wherein the two-dimensional ear representation comprises arepresentation of the at least one biometric feature of the ear.
 4. Themethod according to claim 3, wherein: said representation of the atleast one biometric feature of the ear is determined from atwo-dimensional ear image corresponding to said one of the plurality ofpre-stored three-dimensional ear shapes.
 5. The method according toclaim 2, wherein the two-dimensional ear representation comprises atwo-dimensional ear image corresponding to said one of the plurality ofpre-stored three-dimensional ear shapes.
 6. The method according toclaim 1, wherein the matching comprises: determining that a property ofsaid at least one biometric feature of the ear is within a tolerance ofa corresponding property of said one of the plurality of pre-storedthree-dimensional ear shapes.
 7. The method according to claim 1,wherein said at least one biometric feature of the ear comprises atleast one measurement of a feature of the ear.
 8. The method accordingto claim 7, wherein the at least one measurement of a feature of the earcomprises a measurement of a helix curvature of the ear.
 9. The methodaccording to claim 7, wherein the at least one measurement of a featureof the ear comprises a measurement of at least one triangle defined bythree predefined anatomical points of the ear.
 10. The method accordingto claim 9, wherein the three predefined anatomical points are locatedon: the helix of the ear; the Fossa triangularis of the ear; and thelobe of the ear.
 11. The method according to claim 9, wherein the threepredefined anatomical points of a triangle of the at least one triangleare located on: the intertragic notch of the ear; the Fossa triangularisof the ear; and the antihelix of the ear.
 12. The method according toclaim 9, wherein the three predefined anatomical points of a triangle ofthe at least one triangle are located on: the intertragic notch of theear; the tragus of the ear; and the antitragus of the ear.
 13. Themethod according to claim 9, wherein the three predefined anatomicalpoints of a triangle of the at least one triangle are located on: theantihelix of the ear; the tragus of the ear; and the helix of the ear.14. The method according to claim 1, wherein: each of the plurality ofpre-stored three-dimensional shapes is produced by three-dimensionallyscanning a respective ear canal.
 15. The method according to claim 1,further comprising: manufacturing an earbud having a shape correspondingto the three-dimensional shape of the insert.
 16. The method accordingto claim 1, further comprising: constraining the matching according toat least one matching constraint.
 17. The method according to claim 1,further comprising: identifying an object of known spatial dimension inthe two-dimensional image; and calculating a scale of thetwo-dimensional image relative to the object of known spatial dimension.18. A server adapted to produce an ear insert, the insert having athree-dimensional shape which is derived from image data correspondingto a two-dimensional image of an ear, the server comprising: a receivingmodule configured to receive at the server from a remote device, theimage data corresponding to the two-dimensional image of the ear, thereceiving module determining whether the image data received by theserver satisfies image quality criteria, the receiving module configuredto reject the image data by the server if the received image data doesnot satisfy the image quality criteria and to send a signal from theserver to the remote device requesting replacement image data; ameasuring module configured to scale and re-orient, at the server, theimage data, to be within a predetermined acceptable range for theserver; an extraction module configured to extract, at the server,particular anatomical features and make associated measurements from theimage data to obtain at least one biometric feature of the ear, the atleast one biometric feature of the ear being representative of athree-dimensional shape of at least part of the ear; and a matchingmodule configured to determine, at the server, the three-dimensionalshape by matching said at least one biometric feature of the ear withone of a plurality of pre-stored three-dimensional ear shapes, whereineach pre-stored three-dimensional ear shape corresponds to a respectiveear.
 19. A system comprising the server according to claim 18, furthercomprising: a database for storing the pre-stored three-dimensional earshapes.
 20. A system comprising the server according to claim 19,further comprising: a manufacturing apparatus configured to manufacturean earbud with a shape corresponding to the three-dimensional ear shape.21. The system according to claim 20, wherein the server is connected toa three-dimensional printing, or additive manufacturing, system.